In [1]:
__author__ = 'Stephanie Juneau <stephanie.juneau@noirlab.edu>, David Herrera <david.herrera@noirlab.edu>, and the Astro Data Lab Team <datalab@noirlab.edu>'
__version__ = '20240606' # yyyymmdd
__datasets__ = ['ls_dr10', 'sdss_dr17']
__keywords__ = ['extragalactic', 'galaxies', 'joint query', 'spectroscopic redshift', '3d plot']

DESI Legacy Surveys and SDSS/BOSS Large Scale Structure¶

by Stéphanie Juneau, David Herrera, and the Astro Data Lab Team

Table of contents¶

  • Goals & Summary
  • Disclaimer & attribution
  • Imports & setup
  • Joint Query of LS and SDSS catalogs
  • Plot Results
  • Exercise
  • Large scale structure in 3D

Goals¶

  • Joint query between photometric (LS) and spectroscopic (SDSS) catalogs
  • Plot on-sky position of extragalactic objects, color-coded by redshift
  • Plot positions of galaxies in 3D (real space), and explore large scale structure interactively

Summary¶

In this Notebook, we explore large-scale structures of galaxies by combining spectroscopic redshifts from SDSS/BOSS with photometry from the DESI pre-imaging Legacy Survey (LS). The advantage of spectroscopic redshifts is that they are far more accurate than photometric redshifts to probe distances to galaxies (though still need to be corrected for possible distortion effects, which we ignore here). The advantage of the LS photometry is that it reaches deeper than SDSS by about 1 magnitude, which yields better image quality to measure magnitudes, colors, and galaxy shapes. While there are several possible extensions to the example work included below, we will show that a simple figure of galaxy spatial locations color-coded by galaxy morphological type reveals the known morphology-density relation.

We wanted to extend indeed a little further and be able to visualize and even interact with a representation of these galaxies in the actual space. For that, we developed a 3D plot based on converting RA, DEC and z, into Cartesian coordinates as shown at the end of this notebook.

On a technical point of view, this short notebook illustrates an example joint query between the LS DR10 photometry Tractor table, and the SDSS/BOSS DR17 specObj spectroscopy table. It uses a pre-crossmatched table based on the closest match within a 1.5 arcsec search radius.

The columns from the LS table used (Tractor) can be seen here: https://datalab.noirlab.edu/query.php?name=ls_dr10.tractor

The columns from the pre-crossmatched table can be seen here: https://datalab.noirlab.edu/query.php?name=ls_dr10.x1p5__tractor__sdss_dr17__specobj

The columns from the SDSS DR17 used for the 3D plot can be found here https://datalab.noirlab.edu/query.php?name=sdss_dr17.specobj

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024

https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

Imports and setup¶

Please note that this notebook is written for Python 3.

In [2]:
# std lib
from getpass import getpass

# 3rd party
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from scipy.stats import binned_statistic_2d
%matplotlib inline
from astropy.table import Table
from astropy.cosmology import Planck18 as cosmo
import plotly
import plotly.graph_objs as go
import pandas as pd
plotly.offline.init_notebook_mode()

# Data Lab
from dl import queryClient as qc
from dl import authClient as ac

Authentication¶

Much of the functionality of Data Lab can be accessed without explicitly logging in (the service then uses an anonymous login). But some capacities, for instance saving the results of your queries to your virtual storage space, require a login (i.e. you will need a registered user account).

If you need to log in to Data Lab, un-comment the cell below and execute it:

In [3]:
#token = ac.login(input("Enter user name: (+ENTER) "),getpass("Enter password: (+ENTER) "))
ac.whoAmI()
Out[3]:
'i_olin'

Query LS Tractor Photometry Catalog¶

The photometry is derived from Tractor modeling of sources, and the database includes model photometry, type (shape), as well as other quantities.

The Legacy Survey DR10 database is called ls_dr10 and includes several tables. We will use the tractor table together with the positional crossmatched table with specObj table from SDSS/BOSS DR17. The column names and descriptions can be found directly in a notebook with qc.schema() or from the Data Lab Query Interface or using the Table Access Protocol (TAP) service with a compatible tool such as TOPCAT.

The SDSS DR17 database is called sdss_dr17 and also includes several tables. We will use the specobj table, which has spectroscopic information.

In [4]:
# number of rows from LS DR10 tractor (NOTE: tractor is the main photometry table):
query="SELECT nrows FROM tbl_stat WHERE schema='ls_dr10' and tbl_name='tractor'"

# Call query manager
response = qc.query(sql=query, fmt='csv')

print(response)

nrows
3147090432
In [5]:
# number of rows from SDSS specObj DR17:
query="SELECT count(*) FROM sdss_dr17.specobj"

# Call query manager
response = qc.query(sql=query, fmt='csv')

print(response)

count
5112724
In [6]:
# Print columns for the LS DR10 Tractor table
print(qc.schema('ls_dr10.tractor'))

Schema: ls_dr10
 Table: tractor

     Column Name   Description
     -----------   -----------
         release   Integer denoting the camera and filter set used, which wil
                   l be unique for a given processing run of the data (as doc
                   umented here_)
         brickid   Brick ID [1,662174]
       brickname   Name of brick, encoding the brick sky position, eg 1126p22
                   2 near RA=112.6, Dec=+22.2
           objid   Catalog object number within this brick; a unique identifi
                   er hash is release,brickid,objid;  objid spans [0,N-1] and
                   is contiguously enumerated within each brick
   brick_primary   True if the object is within the brick boundary
        maskbits   Bitwise mask indicating that an object touches a pixel in 
                   the coadd/*/*/*maskbits* maps, as cataloged on the DR10 bi
                   tmasks page_
         fitbits   Bitwise mask detailing pecularities of how an object was f
                   it, as cataloged on the DR10 bitmasks page_
            type   Morphological model: PSF=stellar, REX=round exponential ga
                   laxy, DEV=deVauc, EXP=exponential, SER=Sersic, DUP=Gaia so
                   urce fit by different model. See also the larger descripti
                   on_.
              ra   Right ascension at equinox J2000
             dec   Declination at equinox J2000
         ra_ivar   Inverse variance of RA (no cosine term!), excluding astrom
                   etric calibration errors
        dec_ivar   Inverse variance of DEC, excluding astrometric calibration
                   errors
     flux_ivar_i   Inverse variance of flux_i
              bx   X position (0-indexed) of coordinates in the brick image s
                   tack (*i.e.* in the *e.g.* legacysurvey-<brick>-image-g.fi
                   ts.fz coadd file_)
              by   Y position (0-indexed) of coordinates in brick image stack
        dchisq_1   Difference in |chi|^2 between successively more-complex mo
                   del fits: PSF, REX, DEV, EXP, SER.  The difference is vers
                   us no source.
        dchisq_2   Difference in |chi|^2 between successively more-complex mo
                   del fits: PSF, REX, DEV, EXP, SER.  The difference is vers
                   us no source.
        dchisq_3   Difference in |chi|^2 between successively more-complex mo
                   del fits: PSF, REX, DEV, EXP, SER.  The difference is vers
                   us no source.
        dchisq_4   Difference in |chi|^2 between successively more-complex mo
                   del fits: PSF, REX, DEV, EXP, SER.  The difference is vers
                   us no source.
        dchisq_5   Difference in |chi|^2 between successively more-complex mo
                   del fits: PSF, REX, DEV, EXP, SER.  The difference is vers
                   us no source.
             ebv   Galactic extinction E(B-V) reddening from SFD98_, used to 
                   compute the mw_transmission_ columns
         mjd_min   Minimum Modified Julian Date of observations used to const
                   ruct the model of this object
         mjd_max   Maximum Modified Julian Date of observations used to const
                   ruct the model of this object
         ref_cat   Reference catalog source for this star: T2 for Tycho-2_, G
                   E for Gaia EDR3_, L3 for the SGA_, empty otherwise
          ref_id   Reference catalog identifier for this star; Tyc1*1,000,000
                   +Tyc2*10+Tyc3 for Tycho2; sourceid for Gaia EDR3_ and SGA_
            pmra   Reference catalog proper motion in the RA direction
           pmdec   Reference catalog proper motion in the Dec direction
        parallax   Reference catalog parallax
       pmra_ivar   Reference catalog inverse-variance on pmra
      pmdec_ivar   Reference catalog inverse-variance on pmdec
   parallax_ivar   Reference catalog inverse-variance on parallax
       ref_epoch   Reference catalog reference epoch (eg, 2015.5 for Gaia EDR3_)
gaia_phot_g_mean_mag   Gaia EDR3_ G band mag
gaia_phot_g_mean_flux_over_error   Gaia EDR3_ G band signal-to-noise
gaia_phot_g_n_obs   Gaia EDR3_ G band number of observations
gaia_phot_bp_mean_mag   Gaia EDR3_ BP mag
gaia_phot_bp_mean_flux_over_error   Gaia EDR3_ BP signal-to-noise
gaia_phot_bp_n_obs   Gaia EDR3_ BP number of observations
gaia_phot_rp_mean_mag   Gaia EDR3_ RP mag
gaia_phot_rp_mean_flux_over_error   Gaia EDR3_ RP signal-to-noise
gaia_phot_rp_n_obs   Gaia EDR3_ RP number of observations
gaia_phot_variable_flag   Gaia EDR3_ photometric variable flag
gaia_astrometric_excess_noise   Gaia EDR3_ astrometric excess noise
gaia_astrometric_excess_noise_sig   Gaia EDR3_ astrometric excess noise uncertainty
gaia_astrometric_n_obs_al   Gaia EDR3_ number of astrometric observations along scan d
                   irection
gaia_astrometric_n_good_obs_al   Gaia EDR3_ number of good astrometric observations along s
                   can direction
gaia_astrometric_weight_al   Gaia EDR3_ astrometric weight along scan direction
gaia_duplicated_source   Gaia EDR3_ duplicated source flag
    gaia_a_g_val   Gaia EDR3_ line-of-sight extinction in the G band
gaia_e_bp_min_rp_val   Gaia EDR3_ line-of-sight reddening E(BP-RP)
gaia_phot_bp_rp_excess_factor   Gaia EDR3_ BP/RP excess factor
gaia_astrometric_sigma5d_max   Gaia EDR3_ longest semi-major axis of the 5-d error ellipsoid
gaia_astrometric_params_solved   Which astrometric parameters were estimated for a Gaia EDR
                   3_ source
          flux_g   model flux in g
          flux_r   model flux in r
          flux_i   model flux in i
          flux_z   model flux in z
         flux_w1   WISE model flux in W1 (AB system)
         flux_w2   WISE model flux in W2 (AB)
         flux_w3   WISE model flux in W3 (AB)
         flux_w4   WISE model flux in W4 (AB)
     flux_ivar_g   Inverse variance of flux_g
     flux_ivar_r   Inverse variance of flux_r
           w1_w2   Computed (w1-w2) color
     flux_ivar_z   Inverse variance of flux_z
    flux_ivar_w1   Inverse variance of flux_w1 (AB system)
    flux_ivar_w2   Inverse variance of flux_w2 (AB)
    flux_ivar_w3   Inverse variance of flux_w3 (AB)
    flux_ivar_w4   Inverse variance of flux_w4 (AB)
     fiberflux_g   Predicted g-band flux within a fiber of diameter 1.5 arcse
                   c from this object in 1 arcsec Gaussian seeing
     fiberflux_r   Predicted r-band flux within a fiber of diameter 1.5 arcse
                   c from this object in 1 arcsec Gaussian seeing
     fiberflux_i   Predicted i-band flux within a fiber of diameter 1.5 arcse
                   c from this object in 1 arcsec Gaussian seeing
     fiberflux_z   Predicted z-band flux within a fiber of diameter 1.5 arcse
                   c from this object in 1 arcsec Gaussian seeing
  fibertotflux_g   Predicted g-band flux within a fiber of diameter 1.5 arcse
                   c from all sources at this location in 1 arcsec Gaussian s
                   eeing
  fibertotflux_r   Predicted r-band flux within a fiber of diameter 1.5 arcse
                   c from all sources at this location in 1 arcsec Gaussian s
                   eeing
  fibertotflux_i   Predicted i-band flux within a fiber of diameter 1.5 arcse
                   c from all sources at this location in 1 arcsec Gaussian s
                   eeing
  fibertotflux_z   Predicted z-band flux within a fiber of diameter 1.5 arcse
                   c from all sources at this location in 1 arcsec Gaussian s
                   eeing
mw_transmission_g   Galactic transmission in g filter in linear units [0, 1]
mw_transmission_r   Galactic transmission in r filter in linear units [0, 1]
mw_transmission_i   Galactic transmission in i filter in linear units [0, 1]
mw_transmission_z   Galactic transmission in z filter in linear units [0, 1]
mw_transmission_w1   Galactic transmission in W1 filter in linear units [0, 1]
mw_transmission_w2   Galactic transmission in W2 filter in linear units [0, 1]
mw_transmission_w3   Galactic transmission in W3 filter in linear units [0, 1]
mw_transmission_w4   Galactic transmission in W4 filter in linear units [0, 1]
          nobs_g   Number of images that contribute to the central pixel in g
                   filter for this object (not profile-weighted)
          nobs_r   Number of images that contribute to the central pixel in r
                   filter for this object (not profile-weighted)
          nobs_i   Number of images that contribute to the central pixel in i
                   filter for this object (not profile-weighted)
          nobs_z   Number of images that contribute to the central pixel in z
                   filter for this object (not profile-weighted)
         nobs_w1   Number of images that contribute to the central pixel in W
                   1 filter for this object (not profile-weighted)
         nobs_w2   Number of images that contribute to the central pixel in W
                   2 filter for this object (not profile-weighted)
         nobs_w3   Number of images that contribute to the central pixel in W
                   3 filter for this object (not profile-weighted)
         nobs_w4   Number of images that contribute to the central pixel in W
                   4 filter for this object (not profile-weighted)
        rchisq_g   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in g
        rchisq_r   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in r
        rchisq_i   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in i
        rchisq_z   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in z
       rchisq_w1   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in W1
       rchisq_w2   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in W2
       rchisq_w3   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in W3
       rchisq_w4   Profile-weighted |chi|^2 of model fit normalized by the nu
                   mber of pixels in W4
      fracflux_g   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in g (typically [0,1])
      fracflux_r   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in r (typically [0,1])
      fracflux_i   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in i (typically [0,1])
      fracflux_z   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in z (typically [0,1])
     fracflux_w1   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in W1 (typically [0,1])
     fracflux_w2   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in W2 (typically [0,1])
      blob_nea_i   Blob-masked noise equivalent area_ in i.
     fracflux_w3   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in W3 (typically [0,1])
     fracflux_w4   Profile-weighted fraction of the flux from other sources d
                   ivided by the total flux in W4 (typically [0,1])
    fracmasked_g   Profile-weighted fraction of pixels masked from all observ
                   ations of this object in g, strictly between [0,1]
    fracmasked_r   Profile-weighted fraction of pixels masked from all observ
                   ations of this object in r, strictly between [0,1]
    fracmasked_i   Profile-weighted fraction of pixels masked from all observ
                   ations of this object in i, strictly between [0,1]
    fracmasked_z   Profile-weighted fraction of pixels masked from all observ
                   ations of this object in z, strictly between [0,1]
        fracin_g   Fraction of a source flux within the blob in g, near unity
                   for real sources
        fracin_r   Fraction of a source flux within the blob in r, near unity
                   for real sources
        fracin_i   Fraction of a source flux within the blob in i, near unity
                   for real sources
        fracin_z   Fraction of a source flux within the blob in z, near unity
                   for real sources
         ngood_g   Number of good (unmasked) images that contribute in g (thi
                   s quantity is consistent with the nexp maps in the image s
                   tacks_)
         ngood_r   Number of good (unmasked) images that contribute in r (thi
                   s quantity is consistent with the nexp maps in the image s
                   tacks_)
         ngood_i   Number of good (unmasked) images that contribute in i (thi
                   s quantity is consistent with the nexp maps in the image s
                   tacks_)
         ngood_z   Number of good (unmasked) images that contribute in z (thi
                   s quantity is consistent with the nexp maps in the image s
                   tacks_)
       anymask_g   Bitwise mask set if the central pixel from any image satis
                   fies each condition in g as cataloged on the DR10 bitmasks
                   page_
       anymask_r   Bitwise mask set if the central pixel from any image satis
                   fies each condition in r as cataloged on the DR10 bitmasks
                   page_
       anymask_i   Bitwise mask set if the central pixel from any image satis
                   fies each condition in i as cataloged on the DR10 bitmasks
                   page_
       anymask_z   Bitwise mask set if the central pixel from any image satis
                   fies each condition in z as cataloged on the DR10 bitmasks
                   page_
       allmask_g   Bitwise mask set if the central pixel from all images sati
                   sfy each condition in g as cataloged on the DR10 bitmasks
                   page_
       allmask_r   Bitwise mask set if the central pixel from all images sati
                   sfy each condition in r as cataloged on the DR10 bitmasks
                   page_
       allmask_i   Bitwise mask set if the central pixel from all images sati
                   sfy each condition in i as cataloged on the DR10 bitmasks
                   page_
       allmask_z   Bitwise mask set if the central pixel from all images sati
                   sfy each condition in z as cataloged on the DR10 bitmasks
                   page_
     wisemask_w1   W1 bitmask as cataloged on the DR10 bitmasks page_
     wisemask_w2   W2 bitmask as cataloged on the DR10 bitmasks page_
       psfsize_g   Weighted average PSF FWHM in the g band
       psfsize_r   Weighted average PSF FWHM in the r band
       psfsize_i   Weighted average PSF FWHM in the i band
       psfsize_z   Weighted average PSF FWHM in the z band
      psfdepth_g   For a 5-sigma point source detection limit in g, 5/(sqrt p
                   sfdepth_g) gives flux in nanomaggies and -2.5[log10(5 / (s
                   qrt psfdepth_g)) - 9] gives corresponding AB magnitude
      psfdepth_r   For a 5-sigma point source detection limit in r, 5/(sqrt p
                   sfdepth_r) gives flux in nanomaggies and -2.5[log10(5 / (s
                   qrt psfdepth_r)) - 9] gives corresponding AB magnitude
      psfdepth_i   For a 5-sigma point source detection limit in i, 5/(sqrt p
                   sfdepth_i) gives flux in nanomaggies and -2.5[log10(5 / (s
                   qrt psfdepth_i)) - 9] gives corresponding AB magnitude
      psfdepth_z   For a 5-sigma point source detection limit in z, 5/(sqrt p
                   sfdepth_z) gives flux in nanomaggies and -2.5[log10(5 / (s
                   qrt psfdepth_z)) - 9] gives corresponding AB magnitude
      galdepth_g   As for psfdepth_g but for a galaxy (0.45s exp, round) dete
                   ction sensitivity
      galdepth_r   As for psfdepth_r but for a galaxy (0.45s exp, round) dete
                   ction sensitivity
      galdepth_i   As for psfdepth_i but for a galaxy (0.45s exp, round) dete
                   ction sensitivity
      galdepth_z   As for psfdepth_z but for a galaxy (0.45s exp, round) dete
                   ction sensitivity
           nea_g   Noise equivalent area_ in g.
           nea_r   Noise equivalent area_ in r.
           nea_i   Noise equivalent area_ in i.
           nea_z   Noise equivalent area_ in z.
      blob_nea_g   Blob-masked noise equivalent area_ in g.
      blob_nea_r   Blob-masked noise equivalent area_ in r.
      blob_nea_z   Blob-masked noise equivalent area_ in z.
     psfdepth_w1   As for psfdepth_g (and also on the AB system) but for WISE W1
     psfdepth_w2   As for psfdepth_g (and also on the AB system) but for WISE W2
     psfdepth_w3   As for psfdepth_g (and also on the AB system) but for WISE W3
     psfdepth_w4   As for psfdepth_g (and also on the AB system) but for WISE W4
   wise_coadd_id   unWISE coadd brick name (corresponding to the, *e.g.*, leg
                   acysurvey-<brick>-image-W1.fits.fz coadd file_) for the ce
                   nter of each object
          wise_x   X position of coordinates in the brick image stack that co
                   rresponds to wise_coadd_id (see the DR9 updates page_ for
                   transformations between wise_x and bx)
          wise_y   Y position of coordinates in the brick image stack that co
                   rresponds to wise_coadd_id (see the DR9 updates page_ for
                   transformations between wise_y and by)
          sersic   Power-law index for the Sersic profile model (type=SER)
     sersic_ivar   Inverse variance of sersic
         shape_r   Half-light radius of galaxy model for galaxy type type (>0)
    shape_r_ivar   Inverse variance of shape_r
        shape_e1   Ellipticity component 1 of galaxy model for galaxy type type
   shape_e1_ivar   Inverse variance of shape_e1
        shape_e2   Ellipticity component 2 of galaxy model for galaxy type type
   shape_e2_ivar   Inverse variance of shape_e2
           ls_id   Unique LS object ID
            htm9   HTM index (order 9 => ~10 arcmin size)
         ring256   HEALPIX index (Nsides 256, Ring scheme => ~14 arcmin size)
        nest4096   HEALPIX index (Nsides 4096, Nest scheme => ~52 arcsec size
            elat   Ecliptic Latitude
            elon   Ecliptic Longitude
            glat   Galactic Latitude
            glon   Galactic Longitude
           mag_g   Converted g magnitude
           mag_r   Converted r magnitude
           mag_i   Converted i magnitude
           mag_z   Converted g magnitude
          mag_w1   Converted W1 magnitude
          mag_w2   Converted W2 magnitude
          mag_w3   Converted W3 magnitude
          mag_w4   Converted W4 magnitude
    dered_flux_g   Dereddened g band flux
    dered_flux_r   Dereddened r band flux
    dered_flux_i   Dereddened i band flux
   dered_flux_w1   Dereddened w1 band flux
   dered_flux_w2   Dereddened w2 band flux
   dered_flux_w3   Dereddened w3 band flux
   dered_flux_w4   Dereddened w4 band flux
    dered_flux_z   Dereddened z band flux
     dered_mag_g   Dereddened g-band magnitude
     dered_mag_r   Dereddened r-band magnitude
     dered_mag_i   Dereddened i-band magnitude
    dered_mag_w1   Dereddened w1-band magnitude
    dered_mag_w2   Dereddened w2-band magnitude
    dered_mag_w3   Dereddened w3-band magnitude
    dered_mag_w4   Dereddened w4-band magnitude
     dered_mag_z   Dereddened z-band magnitude
           snr_g   Signal-to-Noise ratio in g
           snr_r   Signal-to-Noise ratio in r
           snr_i   Signal-to-Noise ratio in i
          snr_w1   Signal-to-Noise ratio in W1
          snr_w2   Signal-to-Noise ratio in W2
          snr_w3   Signal-to-Noise ratio in W3
          snr_w4   Signal-to-Noise ratio in W4
           snr_z   Signal-to-Noise ratio in z
             g_r   Computed (g-r) color
             r_i   Computed (r-i) color
             r_z   Computed (r-z) color
             i_z   Computed (i-z) color
            z_w1   Computed (z-w1) color
           w2_w3   Computed (w2-w3) color
           w3_w4   Computed (w3-w4) color
       random_id   Random ID in the range 0.0 => 100.0
In [7]:
# ls_dr10.tractor                            # LS DR10 tractor photometry
# ls_dr10.x1p5__tractor__sdss_dr17__specobj  # LS DR10 pre-crossmatched to SDSS DR17 specobj
# sdss_dr17.specobj                          # SDSS DR17 specobj

# Write query statement (sql)
query = ("""
  SELECT L.ra, L.dec, L.type, L.sersic, L.g_r, L.r_z,
         S.z, S.ra as plug_ra, S.dec as plug_dec, S.class, S.veldisp, S.veldisperr 
  FROM ls_dr10.tractor AS L
  JOIN ls_dr10.x1p5__tractor__sdss_dr17__specobj AS X ON L.ls_id=X.id1 
  JOIN sdss_dr17.specobj AS S ON X.id2 = S.specobjid 
  WHERE L.ra BETWEEN %s AND %s AND L.dec BETWEEN %s AND %s AND (L.ra_ivar > 0) 
  LIMIT 100000
  """) % (126,131,7.,12.)  # small region

# L.ra, L.dec        = RA, Dec from Legacy Survey (LS) table    
# L.type             = object type (PSF, SIMP, EXP, DEV, COMP)
# L.g_r, L.r_z       = pre-computed g-r and r-z colors from photometry
# S.z                = redshift (z) from SDSS specObj table
# S.plug_ra, dec     = RA, Dec of SDSS fiber from specObj table
# S.class            = Source class (Star, Galaxy, QSO) from SDSS
# S.veldisp, veldisp = velocity dispersion (and error) from SDSS specObj table
#
# WHERE: requirement that RA & Dec coordinates are within a rectangular region

print(query)

  SELECT L.ra, L.dec, L.type, L.sersic, L.g_r, L.r_z,
         S.z, S.ra as plug_ra, S.dec as plug_dec, S.class, S.veldisp, S.veldisperr 
  FROM ls_dr10.tractor AS L
  JOIN ls_dr10.x1p5__tractor__sdss_dr17__specobj AS X ON L.ls_id=X.id1 
  JOIN sdss_dr17.specobj AS S ON X.id2 = S.specobjid 
  WHERE L.ra BETWEEN 126 AND 131 AND L.dec BETWEEN 7.0 AND 12.0 AND (L.ra_ivar > 0) 
  LIMIT 100000
In [8]:
# Call query client and save output as Astropy Table
result = qc.query(adql=query, fmt='table')
In [9]:
# Print length of table (Nb of rows) and the first 10 rows
print(len(result))
result[:10]

8555
Out[9]:
Table length=10
radectypesersicg_rr_zzplug_raplug_decclassveldispveldisperr
float64float64str3float64float64float64float64float64float64str6float64float64
126.00008644935827.034927726600151SER6.01.30862810.76733780.21285242126.000080000000037.0349361GALAXY276.8216615.253322
126.01493537479277.007741109416743SER5.5061591.37680820.779136660.24393715126.014957.0077364GALAXY195.8433717.137558
126.02663081793837.045709718170351SER6.01.49436760.782222750.2591496126.026630000000017.0457033GALAXY218.1049712.376819
126.02417036580487.067103011297897SER0.93140891.13232230.93555830.06391467126.024227.0671055GALAXY144.7776310.631522
126.013537867047.07235585789409SER2.28285880.94811630.701434140.064374804126.013567.0723868GALAXY212.3964410.097903
126.00355065258187.103901614620744PSF0.00.892715450.60748291.2897644126.003577.1038999QSO0.00.0
126.08377628475857.062631211176722SER6.01.58334920.86267280.31347042126.083797.0626316GALAXY273.9194622.274921
126.11029453520177.059559957377497PSF0.00.0135116580.3441622.2276993126.110290000000027.0595677QSO0.00.0
126.13338790107247.042289793306823PSF0.00.288486480.0357437130.0002704559126.133417.0422755STAR0.00.0
126.13860380582237.043977744211515SER2.54630921.942071.18496130.50945634126.138627.0439339GALAXY339.5352866.76817

Plot Results¶

Sanity check: RA, Dec positions from both tables¶

As a sanity check, we overplot the RA, Dec positions from the SDSS table and from the LS table to visually confirm that they overlap as expected.

In [10]:
# convert RA coordinates from [0,360] to [-180,180] 
chgsign = np.where(result['ra'] > 180)
result['ra'][chgsign] = result['ra'][chgsign]-360.
result['plug_ra'][chgsign] = result['plug_ra'][chgsign]-360.

plt.figure(figsize=(9,8))

# plot RA, Dec from LS catalog in red with larger symbols
plt.scatter(result['ra'],result['dec'],s=3,color='red',marker='1')

# overplot RA, Dec from SDSS catalog in blue with smaller symbols
plt.scatter(result['plug_ra'],result['plug_dec'],s=3,color='black',marker='2')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

plt.axis([xmin, xmax, ymin, ymax])
plt.xlim(reversed(plt.xlim())) # flip the x-axis
plt.xlabel("RA (degrees)", fontsize=20)
plt.ylabel("Dec (degrees)", fontsize=20)
plt.show()

Visual Inspection of Large-Scale Structures¶

Plot the positions of a broad range of redshift, and overplot a thin slice in redshift to show possible structures within that slice.

In [11]:
# Select redshift slice
rz = (result['z'] >0.105) & (result['z']<0.125)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=3,color='r',marker='o',alpha=0.25)

# overplot in blue objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=10,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)

# add rectangle to show where we zoom in next panel
ax1.add_patch(patches.Rectangle((128.65-0.25, 8.85-0.2),0.5,0.4,fill=False,color='b'))


## ZOOM IN A SMALLER REGION

# plot all points in red (all redshifts)
ax2.scatter(result['plug_ra'],result['plug_dec'],s=15,color='r',marker='o',alpha=0.3)

# overplot in blue objects in narrow redshift slice
ax2.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=30,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 128.4
xmax = 128.9
ymin = 8.65
ymax = 9.05

ax2.axis([xmin, xmax, ymin, ymax])
ax2.set_xlim(reversed(ax2.set_xlim())) # flip the x-axis
ax2.set_xlabel("RA (degrees)", fontsize=20)
ax2.set_ylabel("Dec (degrees)", fontsize=20)

# add rectangle to show where we zoom in next panel
ax2.add_patch(patches.Rectangle((128.63, 8.81),0.13,0.107,fill=False,color='b'))

plt.show()

Above, the left-hand panel shows a thin redshift slice (0.105 < z < 0.125, black symbols) among objects with redshifts from the spectroscopic SDSS DR17 sample (red symbols). We can see by eye some large-scale filamentary structures and overdensities. The blue rectangle shows a selected region where we zoom in the right-hand panel. On the latter, we further select a smaller region, which we will use in the next cell below.

In [12]:
## ZOOM IN AGAIN OVER AN EVEN SMALLER REGION
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=30,color='r',marker='o',alpha=0.25)

# overplot in black objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=50,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 128.63
xmax = 128.76
ymin = 8.81
ymax = 8.917

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)

## SHOW DECaLS IMAGE (screenshot pre-made but could instead implement image cutout)
im = plt.imread('DECaLS_screenshot_zoomIn_labels.jpg')
ax2.imshow(im)
ax2.axis('off')

plt.show()

The left-hand panel shows the small region enclosed in the blue rectangle that we chose above (right-hand panel). The galaxies in black are in the same narrow redshift slice as defined previously (0.105 < z < 0.125). The right-hand panel is an image cutout of the same region of the sky from the LS sky viewer. The galaxies encircled correspond to the points in black, and some or perhaps most of them likely belong to a galaxy cluster.

Large-Scale Structures with LS Morphologies¶

There are many possible extensions to this work. For instance, one could plot again with symbols coded with object type (from LS) and/or class (from SDSS) and/or velocity dispersion (from SDSS) and/or other quantities. Here, we will start with the object "TYPE" from LS, related to the morphological shapes.

The object shape (2D light profile) is modeled by the Tractor (Lang, Hogg & Mykytyn) as part of the procedure to compute model photometry.

Possible shapes for LS DR10 Morphological Classification:

  • PSF (point spread function: size will vary with the seeing of the observations)
  • REX (“round exponential” galaxies with a variable effective radius)
  • EXP (exponential profile; spiral galaxies)
  • DEV (deVaucouleurs profile; elliptical galaxies)
  • SER (Sersic profile)
  • DUP (set for Gaia sources duplicated by an extended source -- for information only, no flux)
Figure: Images of galaxies including a nearby elliptical galaxy, a nearby spiral galaxy, and a QSO.
In [13]:
# Select redshift slice
rz = (result['z'] >0.105) & (result['z']<0.125)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=3,color='r',marker='o',alpha=0.25)

# overplot in black objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=10,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)


# Select redshift slice and per morphological type

# DEV or early type Sersic with steep index Sersic>=3
rdev = np.where(((result['type']=='DEV')|((result['type']=='SER')&(result['sersic']>=3))) & rz)
# EXP or late type Sersic with index Sersic<2
rexp = np.where(((result['type']=='EXP')|((result['type']=='SER')&(result['sersic']<2))) & rz)
# Intermediate with 2<= Sersic < 3
rint = np.where(((result['type']=='SER')&(result['sersic']>=2)&(result['sersic']<3)) & rz)

# plot all points in red (all redshifts)
ax2.scatter(result['plug_ra'],result['plug_dec'],s=3,color='gray',marker='o',alpha=0.25)

# overplot in objects in narrow redshift slice color-coded blue (EXP or Late type) or red (DEV or Early type)
ax2.scatter(result['plug_ra'][rexp],result['plug_dec'][rexp],s=15,color='b')         # blue = EXP or late-type Sersic
ax2.scatter(result['plug_ra'][rint],result['plug_dec'][rint],s=15,color='orange')    # orange = Intermediate
ax2.scatter(result['plug_ra'][rdev],result['plug_dec'][rdev],s=15,color='r')         # red = DEV or early-type Sersic

plt.axis([xmin, xmax, ymin, ymax])
plt.xlim(reversed(plt.xlim())) # flip the x-axis
plt.xlabel("RA (degrees)", fontsize=20)
plt.ylabel("Dec (degrees)", fontsize=20)
plt.show()

EXERCISE: Large-Scale Structures with Galaxy Colors¶

There are pre-computed colors available. The columns are described here: https://datalab.noirlab.edu/query.php?name=ls_dr10.tractor

Another possibility would be to plot again the galaxies spatial coordinates, but color-coded according to their photometric colors. This is left as an exercise for the user, but feel free to get in touch with the Astro Data Lab Team if you have questions.

Large scale structure in 3D¶

We can explore filaments and clusters of galaxies better if we can plot them in 3D. We turn to a different area of the sky, and will query for a sample of SDSS galaxies in the near to slightly distant universe, and plot in 3D the cone containing them. To avoid conamination by the galactic plane of the Milky Way, we point our search cone at high galactic latitudes.

Data query and preparation¶

We select (mostly) SDSS galaxies within a 10-degree radius around a high galactic latitude direction, (ra,dec) = (160,45) degrees. We limit our search to positive redshifts between 0.02 and 0.3.

In [14]:
# Create the query to fetch the SDSS data from Data Lab:

query = """
SELECT ra,dec,z
FROM sdss_dr17.specobj
WHERE q3c_radial_query(ra,dec,%s,%s,%s)
AND z between 0.02 AND 0.3
""" % (160,45,10)

print (query)

SELECT ra,dec,z
FROM sdss_dr17.specobj
WHERE q3c_radial_query(ra,dec,160,45,10)
AND z between 0.02 AND 0.3

Run the query to fetch the SDSS data from the ls_dr17.specobj table

In [15]:
selection = qc.query(adql=query, fmt='csv')

CPU times: user 34.3 ms, sys: 9.14 ms, total: 43.5 ms
Wall time: 1.54 s

Reformat output into a table

In [16]:
data = Table.read(selection, format='csv')  #Astropy Table
print("Number of galaxies in the sample: %d" % len(data))
data[:5]

Number of galaxies in the sample: 36331
Out[16]:
Table length=5
radecz
float64float64float64
159.2573254.9414630.072250396
159.1750799999999854.9420070.14156331
162.2443954.8897040.14798966
162.5034954.8589970.17074445
162.4223454.8297010.18014126

Compute luminosity distance for every galaxy, using Planck2018 cosmology values

In [17]:
dist = cosmo.luminosity_distance(data['z'])
dist
Out[17]:
$[337.39763,~691.83881,~726.1405,~\dots,~786.81545,~1396.6174,~1409.3111] \; \mathrm{Mpc}$

To plot the positions of each galaxy in true 3D space, we will convert the angular coordinates ra & dec, and the distance coordinate, to Cartesian coordinates X,Y,Z (all measured in Mpc from the coordinate system origin).

In [18]:
def get_cartesian(ra,dec,dist):
    # convert ra and dec to radians, since numpy expects this as arguments to trigonometric functions
    rarad = np.radians(ra)
    decrad = np.radians(dec)
    X = dist * np.sin(decrad) * np.cos(rarad)
    Y = dist * np.sin(decrad) * np.sin(rarad)
    Z = dist * np.cos(decrad)
    return X,Y,Z
In [19]:
X,Y,Z = get_cartesian(data['ra'],data['dec'],dist)

Plotting in 3D¶

To create an interactive 3D plot (one where we can zoom, pan, and rotate the scene), we will use the plotly package. The galaxies in our sample will be plotted using a 3D scatter routine. We also set some overall properties of the plot, such as the size of the markers, and a color map (we color each galaxy redder the further away it is).

In [20]:
trace = go.Scatter3d(
    x = X,
    y = Y,
    z = Z,
    mode = 'markers',
    marker = {
        'size'      : 0.7,
        'opacity'   : 0.5,
        'color'     : dist, 
        'colorscale': 'OrRd'
    }
)

data = [trace]

Next, we define camera location and the layout of the plot

In [21]:
# set up the view point
camera = dict(
    up = dict(x = 0, y = 0, z = 1),
    center = dict(x = 0.2, y = 0, z = 0),
    eye = dict(x = 0.6, y = -0.6, z = -1.0)
)

# set up the plot scene
layout = go.Layout(
    scene = dict(
        xaxis = dict(title = 'X',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
        yaxis = dict(title = 'Y',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
        zaxis = dict(title = 'Z',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
    ),
        scene_camera = camera,
        plot_bgcolor = 'black',
        paper_bgcolor = 'black',
        title = None,
        showlegend = False,
        width = 800,
        height = 800,
        autosize = False,
        margin = {'l':0, 'r':0, 'b':0, 't':0},
        dragmode = 'orbit'
)
In [ ]:
# Draw the plot
plot_figure = go.Figure(data=data, layout=layout)
plot_figure.update_layout()

Explore the scene by:

  • Zooming in and out (mouse scroll wheel up/down, or click mouse middle button and move mouse up/down, or select "zoom" mode via button in upper-right corner & left-click + drag mouse)
  • Panning left/right and up/down (click mouse left button and drag)
  • Change pan/rotate mode by selecting one of the buttons in the upper-right corner of the plot (pan, orbital rotation, turntable rotation)

Observe how the galaxies form compact groups and clusters, and on larger scales form huge filaments. This is the large scale structure of the universe. In details, we can also notice some elongation along the direction of redshift due to redshift space distortions.